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(54) Robot system having image processing function 

(57) A robot system has an image processing func- 
tion capable of detecting position and/or posture of in- 
dividual workplaces randomly arranged in a pile to de- 
termine posture, or posture and position of a robot op- 
eration suitable for the detected position and/or posture 
of the workpiece. Reference models are created from 
two-dimensional images of a reference workpeace cap- 
tured in a plurality of directions by a first visual sensor 
and stored. Also, the relative positions/postures of the 
first visual sensor with respect to the workpiece at the 
respective image capturing, and relative position/pos- 
ture of a second visual sensor to be situated with respect 
to the workpiece are stored. Matching processing be- 
tween an image of a pile of workpieces captured by the 
camera and the reference models are performed and an 
image of a workpiece matched with one reference mod- 
el is selected. A three-dimensional position/posture of 
the workpiece is determined from the image of the se- 
lected workpiece, the selected reference model and po- 
sition/posture information associated with the reference 
model. The position/posture of the second visual sensor 
to be situated for measurement is determined based on 
the determined position/posture of the workpiece and 
the stored relative position/posture of the second visual 
sensor, and precise position/posture of the workpiece is 
measured by the second visual sensor at the deter- 
mined position/posture of the second visual sensor. A 
picking-up operation for picking out a respective work- 
piece from a randomly arranged pile can be performed 
by a robot, based on the measuring results of the second 
visual sensor. 
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Description *' 

[0001 ] The present invention relates to a robot system 
having an image processing function for detecting three- 
dimensional position and posture (orientation) of an ob- 
ject, and in particular to a robot system having an image 
processing function suitable for a bin-picking operation 
for taking out workpiece one by one from a randomly 
arranged pile of workpieces. 
[0002] The operation of taking out an individual work- 
piece from a randomly arranged pile of workpieces or 
an aggregation of workpieces contained in a container 
of a predetermined size, which workpieces have identi- 
cal shapes and different three-dimensional positions/ 
postures, has been performed manually. In storing 
workpieces on a pallet or placing workpieces at a pre- 
determined position in a machine or a device using a 
(dedicated) robot, since it has been impossible to direct- 
ly take out individual workpieces one by one from the 
randomly arranged pile of workpieces by the dedicated 
robot, it has been necessary to rearrange the workpiec- 
es in advance so as to be picked out by the robot. In this 
rearrangement operation, it has been necessary to take 
out an individual workpiece from the pile manually. 
[0003] The reason why individual workpieces having 
identical shapes and different three-dimensional posi- 
tions/postures cannot be picked out by a robot from a 
randomly arranged pile of workpieces or an aggregation 
of workpieces contained in a container is that the posi- 
tion/posture of individual workpieces in the pile or the 
aggregation cannot be recognised, so that a robot hand 
cannot be placed to a suitable position/posture at which 
the robot hand can hold the individual workpiece. 
[0004] An object of the present invention is to provide 
a robot system having an image processing function ca- 
pable of determining an orientation and/or a position of 
a robot for performing an operation on an object by pre- 
cisely detecting three-dimensional position and posture 
of individual objects in a randomly arranged pile or an 
aggregation in a container in a predetermined region, 
which have identical shapes and different three-dimen- 
sional positions/postures. 

[0005] A robot system of the present invention having 
an image processing function comprises a robot, a first 
image capturing device, a memory and a processor. The 
memory stores reference models created according to 
image data of a reference object captured by the image 
capturing device in a plurality of directions, and stores 
information of the capturing directions to be respectively 
associated with the reference models and information 
of orientation of the robot operation with respect to the 
object. The reference object is the object for detection 
or an object having a shape identical to that of the object 
for detection. The processor performs matching 
processing on image data containing an image for the 
object of detection captured by the first image capturing 
device with said reference models to select an image of 
an object matched with one of the reference models, 



and determines orientation, or orientation and position, 
of an operation to be performed by the robot based on 
the selected image of the object, said one reference 
model and information about the capturing direction and 
5 information about the orientation of the robot operation 
with respect to the object associated with said one ref- 
erence model. 

[0006] With a robot system of the present invention, 
a robot operation can be performed on an individual ob- 

io ject in a pile or on an aggregation of plural kinds of ob- 
jects. Inthis case, image data of plural kinds of reference 
objects are captured by the first image data capturing 
device to create the reference models based on the cap- 
tured image data, and relevant information is addition- 

15 ally stored to be associated with each reference model. 
Each of the reference objects is the object of each kind 
of operation or an object having a shape identical to that 
of the object of operation of each kind. The processor 
determines orientatbn, or orientation and position, of 

20 the robot operation based on the image of the object 
and one reference model selected in the matching 
processing, and the information of the kind associated 
with said one. reference model and the information of 
the orientation of the robot operation with respect to the 

25 object associated with said one reference model. 
[0007] The image capturing device may be a camera 
for capturing two-dimensional images and in this case 
the image data of the reference model are captured by 
the camera from a predetermined distance. 

30 [0008] The robot may situate the second image data 
capturing device to have the determined orientation or 
to have the determined orientation and the determined 
position with respect to the object, and the processor 
may process second image data captured by the sec- 

35 ond image capturing device to detect position and/or 
posture of the object with respect to the second image 
data capturing device. 

[0009] The robot may also situate the second image 
data capturing device to have the determined orienta- 

40 tion or to have the determined orientation and the de- 
termined position with respect to the object, so that the 
second image data capturing device is directed to a 
characterising portion of the object, and the processor 
may detect three-dimensional position and/or posture of 

^5 the object based on three-dimensional position of said 
characterising portion obtained by the second image 
capturing device. 

[0010] The first image data capturing device can be 
used as the second image data capturing device. 

50 [001 1] The second image capturing device may com- 
prise a three-dimensional visual sensor of spot-light 
scanning type capable of measuring distance between 
the sensor and an object, or may comprise a structured- 
light unit for irradiating a structured light on an object 

55 and capturing an image of the object including the irra- 
diated light on the object. 

[0012] The robot operation may be an operation for 
picking up at least one object from a plurality of objects 
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whose positions overlap. 

[0013] For a better understanding of the invention, 
and to show how the same may be carried into effect, 
reference will now be made, by way of example, to the 
accompanying drawings, in which:- 

FIG. 1 is a diagram for showing a picking-up oper- 
ation by a robot to take out an individual workpiece 
from a pile of workpieces using an image process- 
ing apparatus according to an embodiment of the 
present invention; 

FIGS. 2a -2d show an example of reference models; 
FIG. 3 is a block diagram of a principal part of a robot 
controller; 

FIG. 4 is a block diagram of the image processing 
apparatus according to an embodiment of the 
present invention; 

FIG. 5 -is a flowchart of the processing for creating 
reference models; 

FIG. 6 is a flowchart of the processing for the pick- 
ing-up operation; 

FIG. 7 is a table showing relative positions/postures 
of the workpiece relative to the first visual senor and 
the positions/postures of the second visual sensor 
to be situated relative to Ihe workpiece in the case 
of four reference models; 

FIG. 8 is a diagram showing an example of scanning 
motion of a visual sensor capable of obtaining dis- 
tance data; 

FIG. 9 is a diagram of the two-dimensional arrange- 
ment data containing distance data as image data 
obtained by the visual sensor; 
FIG. 10 is a flowchart of processing for obtaining 
the two-dimensional arrangement data. 

[0014] A robot system having an image processing 
function according to an embodiment of the present in- 
vention will be described. As shown in FIG. 1, a two- 
dimensional visual sensor 21 such as a CCD camera, 
is attached to a wrist of a robot RB as an image capturing 
device. In this embodiment, an image of a pile of work- 
pieces, which are objects for detection having identical 
shapes and randomly arranged as shown in FIG. 1, is 
captured by the two dimensional visual sensor 21 , and 
orientation, or orientation and position, of an operation 
to be performed by the robot RB is determined based 
on the captured image. Further, rough position and pos- 
ture of the individual workpieces are detected to deter- 
mine the orientation of the robot based on the captured 
image and then precise position and posture of the 
workpiece are detected by the two-dimensional visual 
sensor 21 or a three-dimensional visual sensor 22. 
[001 5] For this purpose, images of a reference object, 
which is one of workpieces W subjected to a picking-up 
operation or an object having a shape identical to that 
of the workpiece W are captured in different directions 
by the image capturing device and reference models are 
created from the image data obtained by the image cap- 



turing and stored in advance. Matching processing be- 
tween the image data obtained by capturing the image 
of the pile of workpieces and the reference models is 
executed to select an image of one workpiece matched 

5 with one of reference models, and the position/posture 
of the selected workpiece is determined based on the 
selected image of the workpiece in the image field of 
view, the selected one of reference modes and the po- 
sition/posture information associated with the selected 

10 one of the reference models. 

[0016] FIG. 3 is a block diagram showing a principal 
part of a robot controller 10 for use in an embodiment of 
the present invention. A main processor 1 , a memory 2 
including a RAM, a ROM and a nonvolatile memory 

is (such as an EEPROM), an interface 3 for a teaching op- 
erating panel, an interface 6 for external devices, an in- 
terlace 7 for an image processing apparatus and a servo 
control section 5 are connected to a bus B. A teaching 
operating panel 4 is connected to the interface 3. 

20 [0017] A system program for supporting basic func- 
tions of the robot RB and robot controller 10 are stored 
in the ROM of the memory 2. Robot operation programs 
and their related determined data which are taught in 
accordance with various operations are stored in the 

26 nonvolatile memory of the memory 2. The RAM of the 
memory 2 is used for temporary storage of data for var- 
ious arithmetical operations performed by the processor 
1. 

[0018] The servo control section 5 comprises servo 

50 controllers Sal to San (n; sum of the number of all the 
axes of the robot including additional movable axes of 
a tool attached to the wrist of the robot), each composed 
of a processor, a ROM, a RAM. etc. Each servo control- 
ler performs position/velocity loop control and also cur- 

35 rent loop control for its associated servomotor for driving 
the axis, to function as a so-called digital servo controller 
for performing loop control of position, velocity and cur- 
rent by software. Each servomotor M1-Mn for driving 
each axis is drivingly controlled according to outputs of 

40 the associated servo controller 5al-5an through the as- 
sociated servo amplifier 5bl-5bn. Though not shown in 
FIG. 3, a position/velocity detector is attached to each 
servomotor Ml-Mn, and the position and velocity of each 
servomotor detected by the associated position/velocity 

45 detector is fed back to the associated servo controller 
5al-5an. To the input/output interface 6 are connected 
sensors of the robot, and actuators and sensors of pe- 
ripheral devices. 

[001 9] FIG 4 is a block diagram of the image process- 
50 jng apparatus 30 connected to an interface 7 of the robot 
controller 1 0. The image processing apparatus 30 com- 
prises a processor 31 to which are connected through 
a bus 42 a ROM 32 for storing a system program to be 
executed by the processor 31 , an image processor 33, 
55 a camera interface 34 connected to a CCD camera 21 
which is a first visual sensor serving as a first image data 
capturing device, a three-dimensional sensor interface 
35 connected to a three-dimensional sensor 22 as a 
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second visual sensor as a second image data capturing 
device, a MDI 36 with a display such as a CRT or a liquid 
crystal display for inputting and outputting various com- 
mands and data, a frame memory 37, a nonvolatile 
memory 38, a RAM 39 for temporary storage of data, a 
communication interface 40 for the robot controller and 
a console interface 41 for a console 43. An image cap- 
tured by the CCD camera 21 is converted into a light 
and shade grey scale image and stored in the frame 
memory 37. The image processor 33 performs image 
processing of images stored in the frame memory 37 on 
demand by the processor 31 so as to recognise an ob- 
ject. The architecture and function of the image process- 
ing apparatus 30 itself is in no way different from the 
conventional image processing apparatus. The image 
processing apparatus 30 of the present invention is dif- 
ferent from the conventional one in that reference mod- 
els as described later are stored in the nonvolatile mem- 
ory 38 and pattern matching processing is performed on 
an image of a pile of workpieces W captured by the first 
visual sensor 21 using the reference models to obtain 
position and posture of a workpiece W and thus orien- 
tation, or orientation and position, of an operation to be 
performed by the robot RB. Further, the image process- 
ing apparatus of the present invention differs from the 
conventional one in that approach orientation/position 
for more precise detection of the position/posture of the 
workpiece W by the second visual sensor of the three- 
dimensional visual sensor 22 is determined based on 
the detected position/posture of the workpiece W using 
the first visual sensor 21. 

[0020] The CCD camera for obtaining two-dimension- 
al images data is well known in the art and thus detailed 
explanation thereof is omitted here. The three-dimen- 
sional visual sensor 22 for detecting a three-dimension- 
al position of an object by irradiating structured light (slit 
beam) on the object is known and used in various fields. 
This type of sensor may be used as the three dimen- 
sional visual sensor 22. Further, a three-dimensional 
visual sensor of a spot-light scanning type as disclosed 
in Japanese Patent Publication No. 7-270137 may be 
used as the three-dimensional visual sensor 22, and the 
summary of such three-dimensional visual sensor is de- 
scribed below. 

[0021] This visual sensor detects a three-dimensional 
position of an object by irradiating a light beam to form 
a light spot on the object for scanning the object in two 
different directions (X direction and Y direction) and by 
detecting the light reflected on the object by a position 
sensitive detector (PSD). The three dimensional posi- 
tion of the object is measured by a calculation using the 
respective inclination angles 9x,0y of mirrors for scan- 
ning and incident positions of the reflected light beam 
on the PSD. 

[0022] Referring to FIGS. 8-10, a method of obtaining 
two-dimensional arrangement data including distance 
data using the three-dimensional visual sensor will be 
explained briefly. 



[0023] The scanning range (measuring range) on an 
object is set in advance, and an inclination angle 8x,0y 
of the mirrors is controlled discretely. As shown in FIG. 
7, the scanning is performed from a point (1 , 1 ) to a point 
5 (1 , n), from a point (2,1) to a point (2, n), from a point 
(m, 1 ) to a point (m, n) on the X-Y plane within the scan- 
ning range, to measure the three-dimensional positions 
of each reflected point on the object. Also, a distance Z 
(i, j) between the sensor and the reflection point (i, j) on 

10 the object is obtained and stored in the RAM38 of the 
image processing apparatus 30. Thus, the image data 
is obtained as two-dimensional arrangement data in- 
cluding the distance data Z (i, j) between the sensor and 
the reflection point on the object, as shown in FIG. g. 

is [0024] FIG. 1 0 is a flowchart of processing to be exe- 
cuted by the processor 31 of the image processing ap- 
paratus 30 for obtaining the image data. 
[0025] First, indexes i and j are respective ly set to " 1" 
(Step 300) and the inclination angle (0x,9y) of the mir- 

20 rors is set to (xl , yl) to direct to the start point (1,1) and 
an irradiation command with the inclination angle is sent 
to the sensor 20 (Steps 301-303). The sensor irradiates 
a light beam with the mirrors set at the inclination angle. 
The signal representing the image captured by the PSD 

25 is sent to the image processing apparatus 30. The proc- 
essor 31 of the image processing apparatus 30 calcu- 
lates the position of the reflection point on the object 
from the signal Irom the PSD and the inclination angle 
(0x,0y) of the mirrors to obtain the distance Z (i, j) be- 

30 tween the sensor and the position of the reflection point 
on the object. This value Z (i, j) is stored in the RAM 38 
as the two-dimensional arrangement data [i, j] (Step 
304, 305). The calculation for obtaining the position of 
the reflection point and the distance Z (i, j) may be per- 

35 formed by the sensor 20. 

[0026] Then, the index i is incrementally increased by 
"1 " and the inclination angle Dx of the mirror for X-axis 
direction scanning is increased by the predetermined 
amount Ax (Step 306, 307). It is determined whether or 

40 not the index i exceeds the set value n (Step 308). If the 
index i does not exceed the set value n, the procedure 
returns to Step 303 and the processing from Step 303 
to Step 308 is executed to obtain the distance Z (i, j) of 
the next point. Subsequently, the processing of Steps 

45 303-308 is repeatedly executed until the index i exceeds 
the set value n to obtain and store the distance Z (i, j) 
of the respective points (1 , 1) to (1, n) shown in FIG. 8. 
[0027] If it is determined that the index i exceeds the 
set value n in Step 308, the index i is set to "1 " and the 

50 index j is incrementally increased by "I" to increase the 
inclination angle 6y of the mirror for Y-axis direction 
scanning (Steps 309-31 1 ). Then, it is determined wheth- 
er or not the index j exceeds the set value m (Step 31 2) 
and if the index j does not exceed the set value m, the 

55 procedure returns to Step 302 to repeatedly execute the 
processing of Step 302 and the subsequent Steps. 
[0028] Thus, the processing from Step 302 to Step 
312 is repeatedly executed until the index j exceeds the 
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set value m. If* the index j exceeds the set value m, the 
points in the measurement range (scanning range) 
shown in FIG. 8 have been measured entirely, the dis- 
tance data Z (1, 1) - Z (m, n) as two-dimensional ar- 
rangement data are stored in the RAM 39 and the image 
data obtaining processing is terminated. A part of the 
image data of two-dimensional arrangements or a plu- 
rality of distance data can be obtained by appropriately 
omitting the measurement of the distance for the index i. 
[0029] The foregoing is a description on the process- 
ing for obtaining two-dimensional arrangement data as 
image data using the visual sensor capable of measur- 
ing the distance. Using the two-dimensbnal arrange- 
ment data obtained in this way as image data, creation 
of reference models and detection of position and pos- 
ture (orientation) of an object can be performed. 
[0030] In order to simplify the explanation, the follow- 
ing description will be made assuming that a CCD cam- 
era is used as the two-dimensional visual sensor 21 and 
the two-dimensional image data obtained by capturing 
image of the object by the CCD camera is used. 
[0031] Processing for creating reference models will 
be explained referring to FIGS. 2a-2d and FIG. 5. FIG. 
5 is a flowchart showing processing for teaching refer- 
ence models to the image processing apparatus 30 ac- 
cording to the present invention. 
[0032] One reference workpiece (one of the workpiec- 
es W as objects of robot operation or a workpiece having 
a three-dimensional shape identical to that of the work- 
piece W) is arranged at a predetermined position with 
predetermined posture for creating reference models. A 
first (0-th) position and orientation (posture) of the CCD 
camera 21 for capturing the image of the object is set, 
and an axis of rotation and rotation angles with respect 
to the first (0-th) position and orientation (posture) are 
set in order to determine the subsequent positions and 
orientations (postures) of the camera 21 from the teach- 
ing operation panel 4 of the robot controller 10. In addi- 
tion, the number of the positions and orientations (pos- 
tures) of the workpiece at which the camera 21 captures 
the image of the object are set. 
[0033] As shown in FIGS. 2a to 2d, in this example, 
images of the reference workpiece are captured from 
four different directions and reference models are cre- 
ated based on the four image data. As shown in FIG. 
2a, an image of the reference workpiece is captured 
from the direction of a Z-axis of a world coordinate sys- 
tem at 0-th position/orientation (posture) to create 0-th 
reference model. For setting the subsequent positions/ 
orientations (postures), an axis perpendicular to an op- 
tical axis of the camera and passing a central point of 
the workpiece (an origin of a world coordinate system 
set to the workpiece) and rotation angles along the ro- 
tation axis are set. Since the optical axis of the camera 
is set parallel to the Z axis of the world coordinate sys- 
tem, an axis parallel to either the X-axis or the Y-axis of 
the world coordinate system, which is perpendicular to 
the Z axis, can be selected and the workpiece is rotated 
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around the rotation axis at the workpiece position. 
[0034] In the example, an axis parallel to the X-axis 
of the world coordinate system is set as the rotation axis, 
and for the position/posture shown in FIG. 2b, the rota- 

5 tion angle of 30 B is set to rotate the camera by 30° with 
respect to the workpiece along the rotation axis. A first 
reference model is created based on the image data of 
the workpiece at the position/orientation (posture) 
shown in FIG. 2b. Similarly, as shown in FIGS. 2c and 

10 2d, the camera 21 is rotated by 60° and 90° ; respective- 
ly, along the rotation axis for capturing images of the 
workpiece to create second and third reference models. 
Only posture (orientation) information may be stored to 
be associated with each reference model in the case 

is where the relative position between the camera and the 
reference workpiece is not changed, as in this example. 
[0035] Further, in this example, rotation angles of zero 
degree, 30 degrees, 60 degrees and 90 degrees are set 
for creating four reference models. The dividing range 

20 of the rotation angles may be set more finely and/or 
range of the rotation angle may be set greater to create 
more reference models for more precise detection ol the 
position/posture of the workpiece. 
[0036] Processing for creating the four reference 

25 models will be explained referring to flowchart of FIG. 5. 
[0037] As described above, the 0-th position/posture 
of the robot at which the camera 20 captures the image 
of the object, and the rotation axis and the rotation an- 
gles with respect to the 0-th position/posture are set in 

30 advance in order to determine the subsequent positions/ 
postures of the reference workpiece, and also the 
number of the subsequent positions/jpostures of the 
workpiece are set For intelligible explanation, it is as- 
sumed that the optical axis of the camera is parallel to 

35 the Y-axis of the world coordinate system and that a po- 
sition where the X-axis and Y-axis coordinate values are 
identical to those of the reference workpiece arranged 
at a predetermined position with a predetermined pos- 
ture and only the Z-axis coordinate value is different 

^0 form that of the position of the reference workpiece 
taught to the robot as the 0-th image capturing position 
for obtaining the 0-th reference model. Further, the po- 
sitions of the robot where the camera is rotated with re- 
spect to the reference workpiece by 30 degrees, 60 de- 

45 grees and 90 degrees along the axis passing the central 
point of the reference workpiece and parallel to the X- 
axis of the world coordinate system are set as the 1st, 
2nd and 3rd image capturing positions, and the number 
N of the image capturing positions is set "4." 

so [0038] When a command for creating reference mod- 
els is inputted from teaching operation panel 4, the proc- 
essor 1 of the robot controller 10 6ets a counter M for 
counting the number of the image capturing to "0" (Step 
100). The robot is operated to have the M-th positionl- 

55 posture and a command for image capturing is output- 
ted to the image processing apparatus 30 (Step 101). 
In response to this command, the image processing ap- 
paratus 30 performs capturing of an image of the refer- 
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ence'toorkpietfe with the camera 20 and the captured 
image data is stored in the frame memory 37. Further, 
relative position/orientation (posture) of the workpiece 
with respect to the camera is obtained and stored in the 
nonvolatile memory 37 as relative position/orientation 
(posture) of M-th reference model (Step 103). This rel- 
ative position/posture is hereinafter referred to as "first- 
sensor relative position/posture". Thus, position/pos- 
ture of the workpiece in a camera coordinate system set 
to the camera is obtained from the position/posture of 
the camera and the position/posture of the reference 
workpiece in the world coordinate system when captur- 
ing the image by the camera, and is stored as the rela- 
tive position/posture of the workpiece with respect to the 
camera (the first-sensor relative position/posture). For 
example, the position/posture of the workpiece in the 
camera coordinate system is stored as [xO, yO, zO, a0, 
po, y0]c, where a, p and y mean rotation angle around 
X-, Y-, Z- axes, and B c" means the camera coordinate 
system. Further, as described above, in the case where 
the position of the camera relative to the workpiece is 
not changed, only the orientation(posture) of the camera 
[xO, yO, zO, aO, p0, t0]c is stored to be associated with 
each reference model as the relative position/posture of 
the first sensor. 

[0039] The relative position/posture of (the tool center 
point of) the robot to be situated with respect to the work- 
piece, or the relative position/posture of the second vis- 
ual sensor 22 to be situated with respect to the work- 
piece is stored to be associated with M-th reference 
model, for a subsequent operation performed by the ro- 
bot, i.e., a precise detection of position/posture of the 
workpiece by the second visual sensor 22. This relative 
position/posture is hereinafter referred to as "work- 
piece-robot (second sensor) relative position/posture". 
Then, a data-captured signal is sent to the robot con- 
troller 10 (Step 104). 

[0040] The workpiece-robot (second sensor) relative 
position/posture is for determining orientation and/or po- 
sition of a subsequent operation of the robot RB, i.e., 
the orientation/position of the robot at which the three- 
dimensional sensor 22 can detect the object or a char- 
acteristic portion of the object, more precisely. Alterna- 
tively, in the case where the precise detection of the 
workpiece by the three-dimensional sensor 22 is omit- 
ted, the workpiece-robot relative position/posture may 
be set to the relative position/posture between the tool 
center point and the workpiece which is most suitable 
for picking up the workpiece by a robot hand. 
[0041] The workpiece-robot (second sensor) relative 
position/posture is stored as position/posture of the tool 
center point or the three-dimensional visual sensor 22 
in a workpiece coordinate system set to the workpiece. 
The following explanation is given assuming that the rel- 
ative position/posture of the three-dimensional visual 
sensor 22 is stored. The workpiece-robot (second sen- 
sor) relative position/posture is represented by an ap- 
proach vector for determining approach position/pos- 



ture of the three-dimensional visual sensor 22 with re- 
spect to the workpiece in precise measuring position/ 
posture of the workpiece W by the three-dimensional 
visual sensor 22. One approach vector may be com- 
5 monly set to reference models created for one kind of 
reference workpiece, since the optimal position/posture 
of the tool center point with respect to the workpiece can 
be determined unequivocally in accordance with a kind 
of the workpiece for precisely measuring the position/ 

10 posture of the workpiece by the three-dimensional vis- 
ual sensor 22 or picking up the workpiece by the robot 
hand. The approach vector may be selected from a plu- 
rality of patterns prepared in advance and may include 
orientation (posture) information only. 

75 [0042] The workpiece-robot (second sensor) relative 
position/posture can be expressed by three compo- 
nents (X, Y, Z)wp for parallel motion and three compo- 
nents (a, p, Y)wp for rotation, where wp means the work- 
piece coordinate system. In this embodiment, the work- 

20 piece-robot (second sensor) relative position/posture is 
commonly set for all the reference models. Further, The 
information to be stored in Steps 103 and 104 may be 
gathered as the relative position/posture of the second 
visual sensor to be situated with respect to the CCD 

25 camera 21 . 

[0043] Upon receipt of the data-captured signal, the 
processor 1 of the robot controller 10 incrementally in- 
creases the value of the counter M by T (Step 105) and 
determines whether or not the value of the counter M is 

30 less than a set value N (=4) (Step 106). If the value of 
the counter M is less than the set value N, the procedure 
returns to Step 101 to move the robot to the M-th image- 
capturing position/poslure. Processing of Steps 101 and 
the subsequent Steps are repeatedly executed until the 

35 value of the counter M equals the set value N(=4). 
[0044] Thus, the reference models are stored in the 
nonvolatile memory 38 and also the first-sensor relative 
position/posture, which is the relative position/posture 
of the first visual sensor 21 and the workpiece W and 

*o the workpiece-robot (second sensor) relative position/ 
posture, which is the position/posture of the second vis- 
ual sensor 22 (or the robot) to be situated with respect 
to the workpiece W are stored in the nonvolatile memory 
38. 

45 [0045] The reference models may be created from a 
part of the image data of the reference object, and may 
be created by processing the image data of the refer- 
ence object. 

[0046] FIG. 7 shows an example of the first -sensor 
50 relative position/posture and the workpiece-robot (sec- 
ond sensor) relative position/posture in the case of four 
reference models. 

[0047] The first-sensor relative position/posture is ex- 
pressed as position/posture of the workpiece W in the 
55 camera coordinate system (X, Y, Z, a, p, y)c set for the 
CCD camera 21 . The relative positions of the workpiece 
W. i.e., the positions of the 0-th to 3rd reference models 
are the same, being expressed as (X, Y, Z)-( 10.50, 
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-20.8b, 50.50)fand, the rotation angles a for 0-th to 3rd 
reference models for the rotation around the X-axis be- 
ing set to 0, 60 and 90 degrees, respectively. The CCD 
camera is rotated around the axis parallel to the X-axis 
of the world coordinate system with a center of rotation 
set to the origin of the workpiece coordinate system, and 
since the X-axis of the camera coordinate system is set 
parallel tothe X-axis of the world coordinate system, on- 
ly the rotation angles a in the camera coordinate system 
are changed. 

[0048] The workpiece-robot (second sensor) relative 
position/posture is expressed as the position/posture of 
the second visual sensor 22 in the workpiece coordinate 
system (X, Y, Z, a, p.. y)wp set to the workpiece, and the 
approach vector (30.5, 20.5, 60.9, 0.0, 0.0, 0.0)wp is set 
to represent constant orientation with respect to the 
workpiece W. 

[0049] The reference models and the relative posi- 
tion/posture of the workpiece W and the camera 20 are 
stored in the nonvolatile memory 38 of the image 
processing apparatus 30. In the above described em- 
bodiment, the reference models are created using a ro- 
bot, although the reference models may be created by 
a manual operation without using a robot. In this case, 
the reference workpiece is arranged within a field of view 
of the camera connected to the image processing ap- 
paratus 30, and the images of the workpiece with differ- 
ent postures are captured by the camera. The reference 
models are created based on the image data and the 
relative positions/postures of the camera and the work- 
piece at the image capturing are manually inputted, and 
are stored with the respective relative positions/pos- 
tures. 

[0050] Hereinafter, a picking-up operation for taking 
out an individual workpiece by a robot from a pile of 
workpieces each having a shape identical to that of the 
reference workpiece will be described, as an example 
of a method of detecting three-dimensional position/ 
posture of an object, using the image processing appa- 
ratus 30 storing the reference models. 
[0051] FIG. 6 is a flowchart of processing for the pick- 
ing-up operation using the reference models. 
[0052] When a picking-up command is inputted into 
the robot controller 1 0 from the teaching operation panel 
4, the processor 1 operates the robot RB to move the 
camera attached to the robot wrist to an image capturing 
position where a pile of workpieces are within a field of 
view of the CCD camera 21 (Step 200). Three-dimen- 
sional position/posture of the camera 21 on the world 
coordinate system at this image capturing position is 
outputted tothe image processing apparatus 30, and an 
image capturing command is outputted (Step 201). Up- 
on receipt of the image capturing.command, the proc- 
essor 31 of the image processing apparatus 30 captures 
an image of the pile of the workpieces W to obtain image 
data of some workpieces W and store the data in the 
frame memory 37 (Step 202). 
[0053] Then, pattern matching processing is per- 



formed for the image data stored in the frame memory 
37 using one of the reference models (first reference 
model) stored in the nonvolatile memory 37 so as to de- 
tect a workpiece W (Step 203). In this pattern matching 

5 processing, matching of the image data of the reference 
model with the image data of workpieces is performed 
on the basis of position, turn and scale. A determination 
I made as to whether or not an object has a matching 
value equal to or greater than the set value (Step 204). 

10 if an object having a matching value equal or greater 
than the set value is not detected, the procedure pro- 
ceeds to Step 205 to determine whether or not the pat- 
tern matching is performed using all the reference mod- 
els (first to fourth reference models). If the pattern 

is matching using all the reference models has not yet 
been achieved, further pattern matching is performed 
using another reference model (Step 206). 
[0054] If it is determined in Step 204 that an object 
having a matching value equal or greater than the set 

20 value with respect to any of the reference models is de- 
tected, the procedure proceeds to Step 207 to perform 
matching processing on the two-dimensional data of the 
detected workpieces W. using every taught mode. In 
Step 208, the reference model having the largest match- 
es ing value in the pattern matching processing is selected, 
and the relative position/posture of the workpiece W with 
respect to the camera 21 is determined based on the 
first-sensor relative position/posture, i.e., the relative 
position/posture of the camera and the reference work- 

30 piece stored for the selected reference model, and po- 
sition, rotation angle and scale of the image of the work- 
piece in the matching processing as well as data of the 
workpiece-robot (second sensor) relative position/pos- 
ture associated with the selected reference model, 

35 which represent the position/posture of the second sen- 
sor 22 to be situated with respect to the workpiece are 
read from the nonvolatile memory 37 (Step 208). 
[0055] The reference model having the highest 
matching value is selected in this embodiment, although 

40 a reference model of the rotation angle of zero degree 
(the O-th reference model) may be selected with prec- 
edence, or an object having the highest expansion rate 
of scale (the object which is nearest tothe camera, i.e. 
located at the top of the pile in this example) may be 

45 selected with precedence. 

[0056] The position and posture (orientation) of the 
detected workpiece on the world coordinate system is 
determined from the position and posture of the camera 
21 in the world coordinate system, which has been sent 

50 in Step 201, and the relative position/posture of the 
workpiece W with respect tothe camera 21, and is out- 
putted (Step 209). Thus, since the relative position/pos- 
ture of the workpiece W. with respect to the camera 21 
is the position/posture of the workpiece W in the camera 

55 coordinate system, the position and posture (orienta- 
tion) of the detected workpiece W in the world coordi- 
nate system is obtained by an arithmetical operation of 
coordinate transformation using the data of the position/ 
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posture of the' workpiece W in the camera coordinate 
system and the position/posture of the camera 20 in the 
world coordinate system (Step 209). 
[0057] The position/posture of the second visual sen- 
sor 22 to be situated for a subsequent operation in the 
world coordinate system is determined based on the de- 
termined position/posture of the detected workpiece W 
in the world coordinate system and the workpiece-robot 
(second sensor) relative position/posture data (ap- 
proach vector) (Step 21 0). The processor 1 operates the 
robot to situate the second visual sensor 22 to have the 
determined position/posture, and outputs a measuring 
command to the second visual sensor 22 (Step 211). 
[0058] Upon receipt of the measuring command, the 
three-dimensional sensor 22 measures a three-dimen- 
sional position/posture of the workpiece W. Since the 
second visual sensor 22 is situated at a suitable posi- 
tion/posture with respect to the workpiece W designated 
by the approach vector, the three-dimensional position/ 
posture of the workpiece W can be precisely measured. 
The processor 31 of the image processing apparatus 30 
outputs the result of measurement to the robot controller 
10 (Step 212). 

[0059] The robot controller 10 operates the robot to 
perform a picking-up operation to grip and hold the de- 
tected workpiece W and move the held workpiece W to 
a predetermined position, based on the result of meas- 
urement by the three-dimensional visual sensor 22 
(Step 21 3). Then, the procedure returns to Step 202 to 
repeatedly execute the processing of Step 202 and sub- 
sequent Steps. 

[0060] When all the workpieces have been picked up 
from the pile of the workpieces, a matching value equal 
to or greater than the set reference value cannot be ob- 
tained in the pattern matching processing for all refer- 
ence models in Steps 203-206, and the picking-up op- 
eration is terminated. 

[0061] In the case where a pile of the workpieces can- 
not fall within the field of view of the camera 20, or in the 
case where it is not necessary to capture an image of a 
workpiece behind other workpieces by changing the ori- 
entation of the camera, the procedure may return to 
Step 200 when it is determined "Yes" in Step 205, to 
move the camera to another position/posture at which 
an image of the objective workpiece can be captured. 
[0062] In the foregoing embodiment, the three-dimen- 
sional sensor 22 is adopted as the second visual sensor 
for precisely detecting the position/posture of the work- 
piece, although the two-dimensional sensor may be 
adopted instead of the three-dimensional sensor. In the 
case where the two-dimensional sensor is adopted as 
the second visual sensor, the second sensor may be 
provided in addition to the first visual sensor, or the first 
visual sensor may function as the second visual sensor 
[0063] The position/posture of the individual work- 
piece in the pile of workpieces is roughly detected by 
the first visual sensor and the precise position/posture 
ol the detected workpiece is detected by the second vis- 



ual sensor situated at the suitable position/posture 
which is nearer to the workpiece W, to improve the pre- 
cision of detection. Therefore, when the first visual sen- 
sor takes place of the second visual sensor, precise po- 

5 sition/posture of the workpiece can be detected by cap- 
turing the image of the detected workpiece W from the 
shorter distance and performing the matching process- 
ing by the CCD camera as the second sensor. 
[0064] Further, in the case where a wide-angle lens is 

10 installed in the CCD camera as the image capturing de- 
vice, for example there is possibility of judging the incli- 
nation angle to be 30 degrees by influence of parallax 
when a workpiece of zero degree inclination is at a cor- 
ner of a field of view of the camera. In such a case, the 

i5 camera may be moved in parallel in accordance with the 
position of the workpiece in the field of view of the cam- 
era to a position right above the workpiece to lose the 
influence of parallax, and at this position the image cap- 
turing processing of Step 201 and the subsequent Steps 

20 in FIG. 6 is performed so that false judgment is prevent- 
ed. 

[0065] Furthermore, in the foregoing embodiment, the 
first visual sensor is mounted on the distal end of the 
robot wrist, although the first visual sensor may be a sta- 

25 tionary camera fixed at a place above the pile of the 
workpieces. In this arrangement, the distance between 
the camera and the workpieces is lengthened to prevent 
the influence of parallax. Also, it is not necessary to op- 
erate the robot to capture the image of the workpieces 

30 by the first visual sensor to shorten the cycle tirhe. 
[0066] In the foregoing embodiment, one kind of ob- 
ject (workpieces) are detected and picked up, although, 
the robot system can be modified to detect and pick up 
plural kinds of objects (workpieces). 

35 [0067] In this case, Ma is the number of reference 
models created from a reference workpiece of a kind A 
and Mb is the number of reference models created from 
a reference workpiece of a kind B and information of the 
workpiece kinds A or B are additionally stored to be as- 

40 sociated with each reference model. Then, matching 
processing is carried out between the captured image 
and the (Ma+Mb) number of reference models to select 
the matched reference model, and the workpiece kind 
Information, in addition to the orientation and/or position 

<5 for the subsequent operation, are determined. In this 
way, it is not necessary to separate the workpieces ac- 
cording to the kind o1 workpieces in advance, so that the 
mixed kinds of workpiece can be picked up one by one 
to reduce the operational cost. 

50 [0068] In the foregoing embodiment, the position/pos- 
ture of the object is roughly determined based on image 
data captured by the first visual sensor and the precise 
posrtion/posture of the detected workpiece is measured 
by the second visual sensor, although, the second im- 

55 age capturing device may be omitted and the orientation 
and/or position of the robot operation can be determined 
using the first image data capturing device only. In this 
case, the relative posit ton/posture of the robot (tool cent- 
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er point) to beVrtuated with respect to the workpiece is 
stored as the workpiece-robot relative position/posture. 
[0069] According to the present invention, a position/ 
posture of an objective workpiece in a randomly ar- 
ranged pile of workpieces of one kind or a plurality of 
kinds, or an aggregation of workpieces of one kind or a 
plurality of kinds gathered in a predetermined region, 
which have different three-dimensional positions/pos- 
tures, is-detected, and an orientation/position for an op- 
eration on the detected workpiece by a robot is deter- 
mined. Further, the second image data capturing device 
measures the position/posture of the workpiece more 
precisely at the determined position/posture (approach 
vector) for the robot operation. Therefore, the robot can 
securely perform a picking-up operation of picking up 
an individual workpiece from such a pile or an aggrega- 
tion. 



Claims 

1. A robot system having an image processing func- 
tion for determining posture, or posture and position 
of a robot operation on an object comprising: 

a robot; 

a first image capturing device; 
a memory storing reference model created ac- 
cording to image data of a reference object cap- 
tured by said image capturing device in a plu- 
rality of different directions, and storing infor- 
mation of the capturing directions to be respec- 
tively associated with said reference models, 
and information of orientation of the robot op- 
eration with respect to the object, said refer- 
ence object being the object for detection or an 
object having a shape identical to that of the 
object for detection; and 
a processor to perform matching processing on 
image data containing an image of the object 
for detection captured by said image capturing 
device with said reference models to select an 
image of an object matched with one of said 
reference models, and to determine orienta- 
tion, or orientation and position, of an operation 
to be performed by the robot based on the se- 
lected image of the object, said one reference 
model and the information about the capturing 
direction and information about the orientation 
of the robot operation with respect to the object 
associated with said one reference model. 

2. A robot system having an image processing func- 
tion for determining posture, or posture and position 
of a robot operation on an object comprising: 

a robot; 

a first image capturing device; 



a memory storing reference model created ac- 
cording to image data of different kinds of ref- 
erence objects captured by said first image 
capturing device, and storing information of the 

5 kinds respectively associated with said refer- 

ence models, and information about orientation 
of the robot operation with respect to the object 
of each kind, each of said reference objects be- 
ing an object of operation of one such kind or 

10 an object having a shape identical to that of the 

object of operation of one such kind; and 
a processor to perform matching processing on 
image data containing an image of the object 
for operation captured by said first image cap- 

is turing device with said reference models to se- 

lect an image of an object matched with one of 
said reference models, and to determine orien- 
tation, or orientation and position of the robot 
operation based on the selected image of the 

20 object, said one reference model, the informa- 

tion about the kind of information associated 
with said one reference model and the informa- 
tion of the orientation about the robot operation 
respect to the object associated with said one 

25 reference model. 

3. A robot system having an image processing func- 
tion according to claim 1 or 2, wherein said refer- 
ence models are obtained from a part of the image 

30 data of the reference object. 

4. A robot system having an image processing func- 
tion according to claim 1 or 2, wherein said refer- 
ence models are obtained by processing the image 

35 data of the reference object. 

5. A robot system having an image processing func- 
tion according to any preceding claim, wherein said 
first image capturing device comprises a camera for 

*o capturing two-dimensional image data. 

6. A robot system having an image processing func- 
tion according to claim 5, wherein said image data 
of the reference object are captured by said camera 

45 from a predetermined distance. 

7. A robot system having an image processing func- 
tion according to any preceding claim, wherein said 
robot system further comprises a second image 

50 capturing device and 

said robot situates said second image data cap- 
turing device to have said determined orienta- 
tion or to have said determined orientation and 
55 said determined position with respecl to the ob- 

ject, and 

said processor processes second image data 
captured by said second image capturing de- 
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vice to detect position and/or posture of the ob- 
ject with respect to said second image data 
capturing device 

8. A robot system having an image processing tunc- $ 
tion according to any one of claims 1 to 6, wherein 
said robot system further comprises a second im- 
age capturing device for obtaining three-dimension- 
al position; 

10 

said robot situates said second image data cap- 
turing device to have said determined orienta- 
tion or to have said determined orientation and 
said determined position with respect to the ob- 
ject, so that said second image data capturing is 
device is directed to a characterising portion of 
the object; and 

said processor detects three-dimensional posi- 
tion and/or posture of the object based on 
three-dimensional position of said characteris- 20 
ing portion obtained by said second image cap- 
turing device. 

9. A robot system having an image processing func- 
tion according to claim 7 or 8, wherein said first im- 25 
age data capturing device is used as said second 
image data capturing device. 

10. A robot system having an image processing func- 
tion according to claim 7, 8 or 9, wherein said sec- 30 
ond image capturing device comprises a three-di- 
mensional visual sensor of spot-light scanning type 
capable of measuring distance between the sensor 
and an object. 

35 

11. A robot system having an image processing func- 
tion according to claim 7, 8 or 9, wherein said sec- 
ond image data capturing device comprises a struc- 
tured-light unit for irradiating a structured light on an 
object and capturing an image of the object includ- 40 
ing the irradiated light on the object. 

12. A robot system having an image processing func- 
tion according to any preceding claim, wherein said, 
robot operation is an operation for picking up at 45 
least one object from a plurality of objects over- 
lapped with each other. 
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